Flexibility in Research Designs in Empirical Software Engineering

Problem outline: It is common to classify empirical research designs as either qualitative or quantitative. Typically, particular research methods (e.g., case studies, action research, experiments and surveys) are associated with one or the other of these types of design. Studies in empirical software engineering (ESE) are often exploratory and often involve software developers and development organizations. As a consequence, it may be difficult to plan all aspects of the studies, and to be successful, ESE studies must often be designed to handle possible changes during the conduct of the study. A problem with the above classification is that it does not cater for flexibility in design. Position: This paper suggests viewing research in ESE along the axis of flexible and fixed designs, which is both orthogonal to the axis of quantitative and qualitative designs, and independent of the particular research method. According to the traditional view of ESE, changes to the research design in the course of a study are typically regarded as threats to the validity of the results. However, viewing the study designs as flexible, practical challenges can provide useful information. The validity of the results of studies with flexible research designs can be established by applying techniques that are traditionally used for qualitative designs. This paper urges an increased recognition of flexible designs in ESE and discusses techniques for establishing the trustworthiness in flexible designs.